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Motivation 


•  Surface  Measurement  Technologies 

-  LADAR,  structured  light,  vision 

-  Tomography 

-  Sonar/ultrasound 

-  Radar 

•  New  computational  capabilities 

•  Applications 

-  Visualization  -  E.g  command  and  Control 

-  Recognition  -  Surveillance 

-  Analysis 
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This  work  addresses  a  changing  technology  landscape.  There  are  lots  of  new  and 
cheaper  technologies  for  measuring  surface  shape,  and  some  older  technologies 
(such  as  MRI  and  tomography)  can  be  treated  as  surface  measurement 
mechanisms  under  certain  circumstances.  More  memory  and  processor  speed 
enable  us  to  deal  with  surfaces  and  compute  interesting  things.  Finally,  there  are 
new  applications,  driven  by  other  technologies  and  capabilities. 


3D  Surface  Reconstruction 


•  Noise 

•  Occlusions 

•  Geometric  Distortion 
(Calibration) 

•  Registration 

"Registration  and  Reconstruction  from 
the  China  Lake  Range  Data",  Whitaker 
and  Juarez- Valdes,  University  of  Utah 
Technical  Report,  2001. 


Data  Courtesy  U.S.  Naval  Air  Warfare 
Center,  China  Lake,  CA 
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An  example  of  NavAir  data  shows  the  issues  with  measuring  surfaces.  This  is 
airborne  LADAR  data  taken  from  about  1km.  It  has  noise  (random  and 
systematic)  and  it  has  occlusions,  so  that  even  “rectification”  does  not  produce  a 
full  3D  model.  A  reconstruction  system  should  combine  several  such  scans  from 
different  views  and  create  a  full  3D  model. 


Research  Strategy 

•  Establishing  the  relationships  between  measured 
data  and  surfaces 

-  Statistical  foundation-estimation  theory 

-  Estimating  surfaces  directly  form  ladar/sonar/radar 

•  New  priors  -  methods  for  smoothing/processing 
surfaces 

-  Generalizing  image  processing  to  surfaces 

•  Surface  representations 

-  Extending  level  set  technology  for  surface  modeling 

-  New  algorithms/computational  mechanism 
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This  is  how  we  structured  our  research.  Some  theory  relating  surfaces  to  data. 
Some  better  ways  of  filtering  or  smoothing  surfaces.  And  some  new  ways  of 
representing  surfaces  and  performing  the  associated  computation. 


An  example  of  how  this  type  of  capability  could  be  used.  Reconnaissance. 
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Major  Results 

•  Surface  estimation 

-  Statistical  formulation  for  surface  reconstruction 

•  Ladar  and  tomography 

•  Freeform  surfaces  and  parametric  models 

•  Surface  processing  and  analysis 

—  Higher-order  priors  for  surface  reconstruction 

-  General  framework  for  mapping  image  processing 
algorithms  to  surface 

•  Level-set  surface  modeling 

-  New  algorithms  for  fast  propogation/tracking  of 
interfaces 

-  New  computational  scheme  for  higher-order  flows 
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Here  are  the  results  that  came  out  of  this  project.  All  of  these  results  have 
associated  published  papers  and  software. 


Surface  Estimation 

A  Systems  Approach 


X  A(X)  B(X) 


Bayesian  reconstruction:  maximizing  the  posterior  -  likelihood  (data)  +  prior 
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This  is  the  big  picture  of  how  to  formulate  the  problem  of  “estimating”  surfaces 
from  noisy  data.  This  is  an  extension  of  a  philosophy  which  has  been  pursued 
with  some  success  in  the  image  processing  literature.  This  is  a  Bayesian 
formulation  where  the  surface  is  meant  to  optimize  the  statistical  relationship 
between  the  data  (likelihood)  and  what  we  know  about  shapes  in  the  world 
(prior). 


Surface  Estimation  from  Ladar 


Scanner  Model 

-  Geometry 

-  Lines  of  sight 

-  Pose 

Sensor  Model 

-  Noise 

-  Gaussian  w/outliers 

Shape  Optimization 

-  Parameters 

-  Free-form 
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This  is  the  “sensor  model”  it  describes  how,  given  a  surface,  a  particular 
measurement  would  be  produced.  A  typical  ladar  (laser  range  finder)  takes  a  2D 
array  of  measurements.  Each  measurement  corresponds  to  a  line  of  sight  and 
gives  a  distance  to  the  nearest  surface,  with  some  random  noise.  Studies  of  such 
instruments  [Gregor  and  Whitaker,  2002]  show  that  a  Gaussian  noise  model  with 
outliers  is  a  good  model.  If  we  assume  the  measurements  are  independent,  then 
we  can  quanify  the  “likelihood”  of  a  particular  surface,  conditional  on  a  set  of 
range  measurements. 


Estimating  3D  Surface  Parameters 
Via  Maximum  Likelihood 


•  Converges  more  reliably  and  more  aceurately  than  ICP. 

•  Unbiased! 
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If  we  change  the  parameters  of  a  surface  (e.g.  its  orientation  and  size),  we  can  try 
to  find  the  surface  that  gives  the  best  estimate  of  likelihood.  This  is  a  maximum 
likelihood  estimator.  Statistical  theory  says  that  this  estimator  is  “unbiased”, 
which  means  on  the  average,  it  gives  the  correct  answer.  Using  this  estimator  we 
can  align  models  to  range  measurements.  This  slide  shows  some  experiments  in 
2D  to  confirm  that  this  is  a  reliable,  accurate  way  to  fit  surfaces  to  range  data.  It 
does  significantly  better  than  the  conventional  approach  (closest  point  error), 
which  is  biased. 


Surface  Estimation 

Example:  Ladar  Data  From  Indoor  Scene 


Surface  Rendering  of  Single  Scan 


Close  Up  View 
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This  is  some  example  rage  data.  It  is  rendered  as  a  surface.  The  triangles  are 
constructed  by  connecting  nearby  points  in  the  range  image.  This  data  is  noisy. 
A  closeup  of  the  legs  of  the  chair  shows  how  little  information  is  containted 
there. 


One  of  the  big  problems  of  constructing  3D  models  from  range  data  is 
determining  the  relative  position  of  the  scans.  Typically  external  approaches 
such  as  GPS  or  dead  reckoning  are  not  accurate  enough  to  get  good  3D 
reconstructions.  Thus,  people  often  rely  on  data-driven  approaches  for  aligning 
different  scans.  We  can  use  this  maximum-likelihood  approach  for  aligning 
mdoels  to  data  for  registering  one  range  map  to  another.  The  results  proved  to  be 
very  robust,  even  for  complicated  scenes  such  as  that  shown  here  (one  scan 
rendered  in  white  the  other  in  magenta). 
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China  Lake  Data  Registration 
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This  an  example  of  four  different  scans,  taken  from  different  point  of  view, 
registered  into  one  coordinate  system. 


Once  we  have  sets  of  registered  scans,  we  can  fit  free-form  surface  models 
(using  the  same  statistical  approach)  to  multiples  scans  to  get  3D  models  of 
complex  scenes.  In  this  case  the  models  are  represented  volumetrically  as  level 
sets  and  the  surface  estimation  includes  a  prior  (to  be  describe  more  later)  that 
prefers  smooth  surfaces  with  sharp  creases.  This  reconstructed  includes  ten  scans 
from  different  points  of  view. 


Deltasphere  LADAR  Scan 
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This  is  data  from  the  3rd  Tech  Delta  Sphere  scanner. 


A  reconstruction  from  eight  range  images. 


China  Lake  Reconstruction 


Reconstruction  from  the  China  Lake  data  set. 
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Higher-Order  Surface  Processing 
Feature-Preserving  Flows 

•  Image  Processing 

-  E.g.  anisotropic  smoothing  (Perona  &  Malik) 
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Part  of  the  technology  which  makes  this  possible  is  a  generalization  of  ideas  from 
image  processing.  Using  PDEs  that  penalized  gradients  of  the  image,  people 
have  been  able  construct  nonlinear  filters  that  smooth  images  and  enhance  strong 
edges.  These  terms  can  be  used  as  “priors”  and  combined  with  a  likelihood  (data 
fitting)  term  to  do  reconstruction.  One  of  our  goals  was  to  do  this  for  surfaces. 


Strategy 

•  Variational  formulations  of  curvature 

-  Generalization  of  grad  mag  from  IP 

-  Gradient  of  normal  map 

-  Allowance  for  outliers 

•  Decouple  the  normals  from  surface 

-  Process  normals 

-  Refit  surface  to  normal  map 

-  System  of  2nd-order  equations 
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The  strategy  for  surfaces  is  to  construct  energy  functionals  that  rely  on  penalizing 
the  gradients  of  the  normals,  which  live  on  the  surface.  If  this  is  formulated 
correctly,  the  results  will  be  “geometric”,  which  means  they  depend  only  on  the 
shapes  of  the  objects  and  not  the  parameterization  or  coordinate  systems  that  one 
uses.  The  resulting  PDEs  are  not  easy  to  solve,  because  they  include  fourth-order 
derivatives.  We  have  proposed  a  strategy  that  simultaneously  smooths  the 
normals  of  the  surface  while  trying  to  keep  the  surface  aligned  with  those 
normals. 


Surface  Filtering  Strategy 


Surface  &  Normals  Process  Normals 


Refit  Normals 
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This  is  an  example  of  how  this  computational  strategy  works.  A  shape  induces  a 
field  of  normals  which  are  consistent  with  the  surfaces  nearby.  We  can  smooth 
those  normals  (maintaining  unit  length).  Then  we  can  refit  the  surface.  In  this 
case  a  square  would  get  “blurred”  to  a  circle,  which  is  a  steady  state  of  this 
process. 


If  we  use  a  penalty  function  on  the  gradient  of  the  normals  that  allows  for 
outliers,  analogous  to  Peronal&Malik  for  images,  we  obtian  flows  that  smooth 
surfaces  vdiile  enhancing  strong  creases. 


Just  like  with  images,  this  produces  an  anistropic  scale  space,  or  a  series  of 
“cartoon-like”  shapes  which  produces  simpler  objects  that  retain  the  important 
features. 


Surface  Analysis 

•  In  the  context  of  models  for  AR 

•  “It  would  be  great  if  a  soldier  knew  how 
man  steps  it  would  take  to  get  from  building 
A  to  vehicle  B” 

•  Segmentation,  hierarchical  grouping, . . . 

•  Low  level  ->  high  level 
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One  of  the  goals  of  this  work  is  to  produce  segmented  surfaces  that  could  be  used 
for  analysis.  Thus,  using  generalizations  of  ideas  from  image  processing,  we  can 
partition  surfaces  into  meaningful  pieces. 


The  “edges”  of  surface  patches  are  places  where  the  gradient  of  the  normals  (I.e. 
curvature)  is  high. 


Using  watershed  segmentation  we  can  produce  a  hierarchy  of  surface  patches, 
which  can  be  used  to  decompose  a  model  into  its  constituent  parts. 


Watershed  Segmentation 
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Watershed  Segmentation 
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Cramer-Rao  Bounds 

A  Statistical  Estimate  of  Reconstruction  Quality 
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Because  we  are  using  statistical  formulations,  we  can  rely  on  ideas  from  signal 
processing  to  compute  error  bounds.  This  is  a  generalization  of  Cramer-Rao 
bounds,  which  is  typically  used  for  putting  error  bounds  on  parameter  estimates. 
In  this  case  we  have  applied  it  to  shape,  to  give  us  bounds  on  the  position  of  the 
surface  in  the  direction  of  the  normal.  This  ties  back  into  the  sensor  model. 
Thus,  a  particular  range  image,  reduces  the  error  of  the  surface  in  a  way  that 
depends  on  the  noise  in  the  measurements  and  the  direction  from  which  it  is 
taken. 


Processing  Height  Fields 

•  Formulation  of  higher-order  smoothing  for 
2.5D  surfaces 

•  Surface  deformation  gives  PDE  on  image 

•  Appropriate  for  terrain  processing 

•  Preprocessing  +  detection 
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Some  applications  done  require  full  3D  surfaces,  but  rather  height  fields,  in 
which  one  parameter  is  a  function  of  the  other  two.  This  is  useful,  for  instance, 
in  dealing  with  terrain  maps. 


Terrain  Analysis 


Contour-Line  Vertices  -  Ridges  and  Valleys 


Zero  crossing  of 
third-order 
differential  quantities 

Noise  and  small-scale 
structure  dominates 

Generally  difficult  to 
tune  blurring, 
thresholds,  etc. 

Too-many  features  + 
broken  lines 


For  instance,  one  of  the  things  people  want  to  do  with  terrain  data  is  detect 
features,  such  as  valleys  and  ridges.  This  is  useful,  for  instance,  for 
compression.  Valleys  and  ridges  are  indicated  by  high  curvature  in  the  contour 
lines.  Detecting  these  requires  third-order  deratives,  which  are  noisy. 
Preprocessing  can  help  this. 


Anisotropic  Diffusion  for 
Preserving  Ridges  and  Valleys 


The  same  ideas  used  for  level-set  surfaces,  which  smooth  data  while  preserving 
creases,  apply  to  height  fields,  improving  the  detection  of  features. 


Overcoming  The  Computational  Burden 
Interactive,  3D,  Deformable  Models 

•  Nonlinear  PDE  -  Iterative  over  volume 

•  Algorithms:  front-tracking  schemes 

•  Hardware; 

-  Clusters  and  multiprocessors 

•  Scalable 

-  Commodity  graphics  cards 

•  Price  to  performance 

•  Rate  of  improvement  -  specialized  processors 
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Many  of  the  methods  developed  in  this  work  require  solutions  to  nonlinear  PDEs, 
which  can  be  computationally  expensive.  Part  of  the  work  is  the  development  of 
methods  for  solving  these  problems  more  quickly.  We  have  researched  two 
approaches,  one  using  multiprocessors  on  large  super  computers  (e.g.  SGI  Origin 
series)  and  the  other  using  comodity  gr^hics  cards  (GPUs).  Both  approaches 
have  been  sucessful,  but  the  GPU  approach  is  more  practical  for  a  wider  range  of 
potential  applications. 


Motivation 


Solution:  Graphics  Processor 

•  Why? 

-  Data  level  parallelism 

-  High  speed  memory 

-  Computation  power  going  up  quickly 

-  Inexpensive  ($400  vs.  $50,000+) 

•  Issues 

-  GPUs  -  SIMD  processors,  limited  capabilities 

-  Programming  interfaces  primitive 

-  Limited  GPU-CPU  bandwidth/speed 
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GPUs  are  cheap  and  their  power  is  increasing  rapidly.  They  represent  an 
example  of  a  “streaming  architecture”,  which  privides  homogeneous,  fine¬ 
grained  parallelism  (SIMD)  with  very  high  bandwidth  to  memory.  Thus  they  are 
particularly  well  suitedfor  solving  PDEs  on  large  grids. 


This  video  shows  a  slice  of  a  3D  model  as  it  evolves  on  the  GPU.  The  greyscales 
indicated  curvature  values  and  the  little  squares  are  “tiles”  which  allow  us  to 
focus  the  GPU  processing  only  on  those  areas  of  the  computational  domain  that 
are  near  the  moving  wavefront,  which  is  associated  with  the  level-set  PDE. 


34 


This  is  an  demo  of  an  application  for  3D  segmentation  using  level-set  based 
deformable  models.  The  audio  describes  the  specifics. 
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Smoothing  via  Anisotropic  Diffusion  of  Normals”,  Proc.  IEEE  Visualization 
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Whitaker,  ONR  Sept.  19,  2003 


Tech  Transfer  -  Software 

•  VISPack 

-  C-H-,  object-oriented  toolkit 

-  Image/volme  processing,  level-set  solvers 

-  Several  hundred  downloads,  mostly  academics 

•  Insight  Toolkit 

-  NIH/NLM  Sponsored 

-  First  release  this  fall  -  thousands  of  downloads/month 

-  Ported  most  of  the  level-set  and  reconstruction 
algorithms  developed  under  ONR  project 

-  E.g.  recent  contract  for  fourth-order  surface  flows 
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Workshops  That  Presented  This  Work 


•  “Beyond  Blobs”  SIGGRAPH,  2003 

•  “Image  Processing  for  Volume  Graphics”  SIGGRAPH  2002 

•  “Beyond  Blobs”  SIGGRAPH,  2002 

•  “PDEs  for  Graphics  and  Image  Processing”  SIGGFRAPH,  2002 

•  “Image  Processing  for  Volume  Graphics”  IEEE  Visualization, 
2002 

•  “Image  Processing  for  Volume  Graphics”  SIGGRAPH  2001 

•  “Image  Processing  for  Volume  Graphics”  IEEE  Visualization, 
2001 
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